Utility of human judgment ensembles during times of pandemic uncertainty: A case study during the COVID-19 Omicron BA.1 wave in the USA

Responding to a rapidly evolving pandemic like COVID-19 is challenging, and involves anticipating novel variants, vaccine uptake, and behavioral adaptations. Human judgment systems can complement computational models by providing valuable real-time forecasts. We report findings from a study conducted on Metaculus, a community forecasting platform, in partnership with the Virginia Department of Health, involving six rounds of forecasting during the Omicron BA.1 wave in the United States from November 2021 to March 2022. We received 8355 probabilistic predictions from 129 unique users across 60 questions pertaining to cases, hospitalizations, vaccine uptake, and peak/trough activity. We observed that the case forecasts performed on par with national multi-model ensembles and the vaccine uptake forecasts were more robust and accurate compared to baseline models. We also identified qualitative shifts in Omicron BA.1 wave prognosis during the surge phase, demonstrating rapid adaptation of such systems. Finally, we found that community estimates of variant characteristics such as growth rate and timing of dominance were in line with the scientific consensus. The observed accuracy, timeliness, and scope of such systems demonstrates the value of incorporating them into pandemic policymaking workflows.

Predictive performance of multi-model ensemble forecasts of COVID-19 across European nations

BackgroundShort-term forecasts of infectious disease burden can contribute to situational awareness and aid capacity planning. Based on best practice in other fields and recent insights in infectious disease epidemiology, one can maximise the predictive performance of such forecasts if multiple models are combined into an ensemble. Here we report on the performance of ensembles in predicting COVID-19 cases and deaths across Europe between 08 March 2021 and 07 March 2022. MethodsWe used open-source tools to develop a public European COVID-19 Forecast Hub. We invited groups globally to contribute weekly forecasts for COVID-19 cases and deaths reported from a standardised source over the next one to four weeks. Teams submitted forecasts from March 2021 using standardised quantiles of the predictive distribution. Each week we created an ensemble forecast, where each predictive quantile was calculated as the equally-weighted average (initially the mean and then from 26th July the median) of all individual models predictive quantiles. We measured the performance of each model using the relative Weighted Interval Score (WIS), comparing models forecast accuracy relative to all other models. We retrospectively explored alternative methods for ensemble forecasts, including weighted averages based on models past predictive performance. ResultsOver 52 weeks we collected and combined up to 28 forecast models for 32 countries. We found a weekly ensemble had a consistently strong performance across countries over time. Across all horizons and locations, the ensemble performed better on relative WIS than 84% of participating models forecasts of incident cases (with a total N=862), and 92% of participating models forecasts of deaths (N=746). Across a one to four week time horizon, ensemble performance declined with longer forecast periods when forecasting cases, but remained stable over four weeks for incident death forecasts. In every forecast across 32 countries, the ensemble outperformed most contributing models when forecasting either cases or deaths, frequently outperforming all of its individual component models. Among several choices of ensemble methods we found that the most influential and best choice was to use a median average of models instead of using the mean, regardless of methods of weighting component forecast models. ConclusionsOur results support the use of combining forecasts from individual models into an ensemble in order to improve predictive performance across epidemiological targets and populations during infectious disease epidemics. Our findings further suggest that median ensemble methods yield better predictive performance more than ones based on means. Our findings also highlight that forecast consumers should place more weight on incident death forecasts than incident case forecasts at forecast horizons greater than two weeks. Code and data availabilityAll data and code are publicly available on Github: covid19-forecast-hub-europe/euro-hub-ensemble.

Impact of SARS-CoV-2 vaccination of children ages 5-11 years on COVID-19 disease burden and resilience to new variants in the United States, November 2021-March 2022: a multi-model study

BackgroundSARS-CoV-2 vaccination of persons aged 12 years and older has reduced disease burden in the United States. The COVID-19 Scenario Modeling Hub convened multiple modeling teams in September 2021 to project the impact of expanding vaccine administration to children 5-11 years old on anticipated COVID-19 burden and resilience against variant strains. MethodsNine modeling teams contributed state- and national-level projections for weekly counts of cases, hospitalizations, and deaths in the United States for the period September 12, 2021 to March 12, 2022. Four scenarios covered all combinations of: 1) presence vs. absence of vaccination of children ages 5-11 years starting on November 1, 2021; and 2) continued dominance of the Delta variant vs. emergence of a hypothetical more transmissible variant on November 15, 2021. Individual team projections were combined using linear pooling. The effect of childhood vaccination on overall and age-specific outcomes was estimated by meta-analysis approaches. FindingsAbsent a new variant, COVID-19 cases, hospitalizations, and deaths among all ages were projected to decrease nationally through mid-March 2022. Under a set of specific assumptions, models projected that vaccination of children 5-11 years old was associated with reductions in all-age cumulative cases (7.2%, mean incidence ratio [IR] 0.928, 95% confidence interval [CI] 0.880-0.977), hospitalizations (8.7%, mean IR 0.913, 95% CI 0.834-0.992), and deaths (9.2%, mean IR 0.908, 95% CI 0.797-1.020) compared with scenarios where children were not vaccinated. This projected effect of vaccinating children 5-11 years old increased in the presence of a more transmissible variant, assuming no change in vaccine effectiveness by variant. Larger relative reductions in cumulative cases, hospitalizations, and deaths were observed for children than for the entire U.S. population. Substantial state-level variation was projected in epidemic trajectories, vaccine benefits, and variant impacts. ConclusionsResults from this multi-model aggregation study suggest that, under a specific set of scenario assumptions, expanding vaccination to children 5-11 years old would provide measurable direct benefits to this age group and indirect benefits to the all-age U.S. population, including resilience to more transmissible variants.

IMPACT OF WEEKNIGHT AND WEEKEND CURFEWS USING MOBILITY DATA: A CASE STUDY OF BENGALURU URBAN

Karnataka imposed weeknight and weekend curfews to mitigate the spread of the Omicron variant of SARS-CoV-2. We attempt to assess the impact of curfew using community mobility reports published by Google. Then, we quantify the impact of such restrictions via a simulation study. The pattern of weeknight and weekend curfew, followed by relaxations during the weekdays, seems, at best, to slow and delay the Omicron spread. The simulation outcomes suggest that Omicron eventually spreads and affects nearly as much of the population as it would have without the restrictions. Further, if Karnataka cases trajectory follows the South African Omicron wave trend and the hospitalisation is similar to that observed in well-vaccinated countries (2% of the confirmed cases), then the healthcare requirement is likely within the capacity of Bengaluru Urban when the caseload peaks, with or without the mobility restrictions. On the other hand, if Karnataka cases trajectory follows both the South African Omicron wave trend and the hospitalisation requirement observed there (6.9%), then the healthcare capacity may be exceeded at peak, with or without the mobility restrictions.

Projected resurgence of COVID-19 in the United States in July-December 2021 resulting from the increased transmissibility of the Delta variant and faltering vaccination

What is already known about this topic?The highly transmissible SARS-CoV-2 Delta variant has begun to cause increases in cases, hospitalizations, and deaths in parts of the United States. With slowed vaccination uptake, this novel variant is expected to increase the risk of pandemic resurgence in the US in July--December 2021. What is added by this report?Data from nine mechanistic models project substantial resurgences of COVID-19 across the US resulting from the more transmissible Delta variant. These resurgences, which have now been observed in most states, were projected to occur across most of the US, coinciding with school and business reopening. Reaching higher vaccine coverage in July--December 2021 reduces the size and duration of the projected resurgence substantially. The expected impact of the outbreak is largely concentrated in a subset of states with lower vaccination coverage. What are the implications for public health practice?Renewed efforts to increase vaccination uptake are critical to limiting transmission and disease, particularly in states with lower current vaccination coverage. Reaching higher vaccination goals in the coming months can potentially avert 1.5 million cases and 21,000 deaths and improve the ability to safely resume social contacts, and educational and business activities. Continued or renewed non-pharmaceutical interventions, including masking, can also help limit transmission, particularly as schools and businesses reopen.

Informing University COVID-19 Decisions Using Simple Compartmental Models

Tracking the COVID-19 pandemic has been a major challenge for policy makers. Although, several efforts are ongoing for accurate forecasting of cases, deaths, and hospitalization at various resolutions, few have been attempted for college campuses despite their potential to become COVID-19 hot-spots. In this paper, we present a real-time effort towards weekly forecasting of campus-level cases during the fall semester for four universities in Virginia, United States. We discuss the challenges related to data curation. A causal model is employed for forecasting with one free time-varying parameter, calibrated against case data. The model is then run forward in time to obtain multiple forecasts. We retrospectively evaluate the performance and, while forecast quality suffers during the campus reopening phase, the model makes reasonable forecasts as the fall semester progresses. We provide sensitivity analysis for the several model parameters. In addition, the forecasts are provided weekly to various state and local agencies.

Evaluating the utility of high-resolution proximity metrics in predicting the spread of COVID-19

AO_SCPLOWBSTRACTC_SCPLOWHigh resolution mobility datasets have become increasingly available in the past few years and have enabled detailed models for infectious disease spread including those for COVID-19. However, there are open questions on how such a mobility data can be used effectively within epidemic models and for which tasks they are best suited. In this paper, we extract a number of graph-based proximity metrics from high resolution cellphone trace data from X-Mode and use it to study COVID-19 epidemic spread in 50 land grant university counties in the US. We present an approach to estimate the effect of mobility on cases by fitting an ODE based model and performing multivariate linear regression to explain the estimated time varying transmissibility. We find that, while mobility plays a significant role, the contribution is heterogeneous across the counties, as exemplified by a subsequent correlation analysis. We subsequently evaluate the metrics utility for case surge prediction defined as a supervised classification problem, and show that the learnt model can predict surges with 95% accuracy and 87% F1-score.

Strategies to Mitigate COVID-19 Resurgence Assuming Immunity Waning: A Study for Karnataka, India

COVID-19 vaccination is being rolled out among the general population in India. Spatial heterogeneities exist in seroprevalence and active infections across India. Using a spatially explicit age-stratified model of Karnataka at the district level, we study three spatial vaccination allocation strategies under different vaccination capacities and a variety of non-pharmaceutical intervention (NPI) scenarios. The models are initialised using on-the-ground datasets that capture reported cases, seroprevalence estimates, seroreversion and vaccine rollout plans. The three vaccination strategies we consider are allocation in proportion to the district populations, allocation in inverse proportion to the seroprevalence estimates, and allocation in proportion to the case-incidence rates during a reference period. The results suggest that the effectiveness of these strategies (in terms of cumulative cases at the end of a four-month horizon) are within 2% of each other, with allocation in proportion to population doing marginally better at the state level. The results suggest that the allocation schemes are robust and thus the focus should be on the easy to implement scheme based on population. Our immunity waning model predicts the possibility of a subsequent resurgence even under relatively strong NPIs. Finally, given a per-day vaccination capacity, our results suggest the level of NPIs needed for the healthcare infrastructure to handle a surge.

Supporting COVID-19 policy response with large-scale mobility-based modeling

Social distancing measures, such as restricting occupancy at venues, have been a primary intervention for controlling the spread of COVID-19. However, these mobility restrictions place a significant economic burden on individuals and businesses. To balance these competing demands, policymakers need analytical tools to assess the costs and benefits of different mobility reduction measures.In this paper, we present our work motivated by our interactions with the Virginia Department of Health on a decision-support tool that utilizes large-scale data and epidemiological modeling to quantify the impact of changes in mobility on infection rates. Our model captures the spread of COVID-19 by using a fine-grained, dynamic mobility network that encodes the hourly movements of people from neighborhoods to individual places, with over 3 billion hourly edges. By perturbing the mobility network, we can simulate a wide variety of reopening plans and forecast their impact in terms of new infections and the loss in visits per sector. To deploy this model in practice, we built a robust computational infrastructure to support running millions of model realizations, and we worked with policymakers to develop an intuitive dashboard interface that communicates our models predictions for thousands of potential policies, tailored to their jurisdiction. The resulting decision-support environment provides policymakers with much-needed analytical machinery to assess the tradeoffs between future infections and mobility restrictions.

All Models Are Useful: Bayesian Ensembling for Robust High Resolution COVID-19 Forecasting

Timely, high-resolution forecasts of infectious disease incidence are useful for policy makers in deciding intervention measures and estimating healthcare resource burden. In this paper, we consider the task of forecasting COVID-19 confirmed cases at the county level for the United States. Although multiple methods have been explored for this task, their performance has varied across space and time due to noisy data and the inherent dynamic nature of the pandemic. We present a forecasting pipeline which incorporates probabilistic forecasts from multiple statistical, machine learning and mechanistic methods through a Bayesian ensembling scheme, and has been operational for nearly 6 months serving local, state and federal policymakers in the United States. While showing that the Bayesian ensemble is at least as good as the individual methods, we also show that each individual method contributes significantly for different spatial regions and time points. We compare our models performance with other similar models being integrated into CDC-initiated COVID-19 Forecast Hub, and show better performance at longer forecast horizons. Finally, we also describe how such forecasts are used to increase lead time for training mechanistic scenario projections. Our work demonstrates that such a real-time high resolution forecasting pipeline can be developed by integrating multiple methods within a performance-based ensemble to support pandemic response. ACM Reference FormatAniruddha Adiga, Lijing Wang, Benjamin Hurt, Akhil Peddireddy, Przemys-law Porebski,, Srinivasan Venkatramanan, Bryan Lewis, Madhav Marathe. 2021. All Models Are Useful: Bayesian Ensembling for Robust High Resolution COVID-19 Forecasting. In Proceedings of ACM Conference (Conference17). ACM, New York, NY, USA, 9 pages. https://doi.org/10.1145/nnnnnnn.nnnnnnn

Prioritizing allocation of COVID-19 vaccines based on social contacts increases vaccination effectiveness

We study allocation of COVID-19 vaccines to individuals based on the structural properties of their underlying social contact network. Even optimistic estimates suggest that most countries will likely take 6 to 24 months to vaccinate their citizens. These time estimates and the emergence of new viral strains urge us to find quick and effective ways to allocate the vaccines and contain the pandemic. While current approaches use combinations of age-based and occupation-based prioritizations, our strategy marks a departure from such largely aggregate vaccine allocation strategies. We propose a novel agent-based modeling approach motivated by recent advances in (i) science of real-world networks that point to efficacy of certain vaccination strategies and (ii) digital technologies that improve our ability to estimate some of these structural properties. Using a realistic representation of a social contact network for the Commonwealth of Virginia, combined with accurate surveillance data on spatio-temporal cases and currently accepted models of within- and between-host disease dynamics, we study how a limited number of vaccine doses can be strategically distributed to individuals to reduce the overall burden of the pandemic. We show that allocation of vaccines based on individuals degree (number of social contacts) and total social proximity time is significantly more effective than the currently used age-based allocation strategy in terms of number of infections, hospitalizations and deaths. Our results suggest that in just two months, by March 31, 2021, compared to age-based allocation, the proposed degree-based strategy can result in reducing an additional 56-110k infections, 3.2-5.4k hospitalizations, and 700-900 deaths just in the Commonwealth of Virginia. Extrapolating these results for the entire US, this strategy can lead to 3-6 million fewer infections, 181-306k fewer hospitalizations, and 51-62k fewer deaths compared to age-based allocation. The overall strategy is robust even: (i) if the social contacts are not estimated correctly; (ii) if the vaccine efficacy is lower than expected or only a single dose is given; (iii) if there is a delay in vaccine production and deployment; and (iv) whether or not non-pharmaceutical interventions continue as vaccines are deployed. For reasons of implementability, we have used degree, which is a simple structural measure and can be easily estimated using several methods, including the digital technology available today. These results are significant, especially for resource-poor countries, where vaccines are less available, have lower efficacy, and are more slowly distributed.

Using Mobility Data to Understand and Forecast COVID19 Dynamics

Disease dynamics, human mobility, and public policies co-evolve during a pandemic such as COVID-19. Understanding dynamic human mobility changes and spatial interaction patterns are crucial for understanding and forecasting COVID-19 dynamics. We introduce a novel graph-based neural network(GNN) to incorporate global aggregated mobility flows for a better understanding of the impact of human mobility on COVID-19 dynamics as well as better forecasting of disease dynamics. We propose a recurrent message passing graph neural network that embeds spatio-temporal disease dynamics and human mobility dynamics for daily state-level new confirmed cases forecasting. This work represents one of the early papers on the use of GNNs to forecast COVID-19 incidence dynamics and our methods are competitive to existing methods. We show that the spatial and temporal dynamic mobility graph leveraged by the graph neural network enables better long-term forecasting performance compared to baselines.

Epidemiological and Economic Impact of COVID-19 in the US

This research measures the epidemiological and economic impact of COVID-19 spread in the US under different mitigation scenarios, comprising of non-pharmaceutical interventions. A detailed disease model of COVID-19 is combined with a model of the US economy to estimate the direct impact of labor supply shock to each sector arising from morbidity, mortality, and lockdown, as well as the indirect impact caused by the interdependencies between sectors. During a lockdown, estimates of jobs that are workable from home in each sector are used to modify the shock to labor supply. Results show trade-offs between economic losses, and lives saved and infections averted are non-linear in compliance to social distancing and the duration of lockdown. Sectors that are worst hit are not the labor-intensive sectors such as Agriculture and Construction, but the ones with high valued jobs such as Professional Services, even after the teleworkability of jobs is accounted for. Additionally, the findings show that a low compliance to interventions can be overcome by a longer shutdown period and vice versa to arrive at similar epidemiological impact but their net effect on economic loss depends on the interplay between the marginal gains from averting infections and deaths, versus the marginal loss from having healthy workers stay at home during the shutdown.

High resolution proximity statistics as early warning for US universities reopening during COVID-19

Reopening of colleges and universities for the Fall semester of 2020 across the United States has caused significant COVID-19 case spikes, requiring reactive responses such as temporary closures and switching to online learning. Until sufficient levels of immunity are reached through vaccination, Institutions of Higher Education will need to balance academic operations with COVID-19 spread risk within and outside the student community. In this work, we study the impact of proximity statistics obtained from high resolution mobility traces in predicting case rate surges in university counties. We focus on 50 land-grant university counties (LGUCs) across the country and show high correlation (PCC > 0.6) between proximity statistics and COVID-19 case rates for several LGUCs during the period around Fall 2020 reopenings. These observations provide a lead time of up to [~]3 weeks in preparing resources and planning containment efforts. We also show how features such as total population, population affiliated with university, median income and case rate intensity could explain some of the observed high correlation. We believe these easily explainable mobility metrics along with other disease surveillance indicators can help universities be better prepared for the Spring 2021 semester.

From 5Vs to 6Cs: Operationalizing Epidemic Data Management with COVID-19 Surveillance

The COVID-19 pandemic brought to the forefront an unprecedented need for experts, as well as citizens, to visualize spatio-temporal disease surveillance data. Web application dashboards were quickly developed to fill this gap, including those built by JHU, WHO, and CDC, but all of these dashboards supported a particular niche view of the pandemic (ie, current status or specific regions). In this paper1, we describe our work developing our own COVID-19 Surveillance Dashboard, available at https://nssac.bii.virginia.edu/covid-19/dashboard/, which offers a universal view of the pandemic while also allowing users to focus on the details that interest them. From the beginning, our goal was to provide a simple visual way to compare, organize, and track near-real-time surveillance data as the pandemic progresses. Our dashboard includes a number of advanced features for zooming, filtering, categorizing and visualizing multiple time series on a single canvas. In developing this dashboard, we have also identified 6 key metrics we call the 6Cs standard which we propose as a standard for the design and evaluation of real-time epidemic science dashboards. Our dashboard was one of the first released to the public, and remains one of the most visited and highly used. Our group uses it to support federal, state and local public health authorities, and it is used by people worldwide to track the pandemic evolution, build their own dashboards, and support their organizations as they plan their responses to the pandemic. We illustrate the utility of our dashboard by describing how it can be used to support data story-telling - an important emerging area in data science.

Predicting Onset of COVID-19 with Mobility-Augmented SEIR Model

Timely interventions and early preparedness of healthcare resources are crucial measures to tackle the COVID-19 disease. To aid these efforts, we developed the Mobility-Augmented SEIR model (MA-SEIR) that leverages Googles aggregate and anonymized mobility data to augment classic compartmental models. We show in a retrospective analysis how this method can be applied at an early stage in the COVID-19 epidemic to forecast its subsequent spread and onset in different geographic regions, with minimal parameterization of the model. This provides insight into the role of near real-time aggregate mobility data in disease spread modeling by quantifying substantial changes in how populations move both locally and globally. These changes would be otherwise very hard to capture using less timely data.

Medical Costs of Keeping the US Economy Open During COVID-19

We use an individual based model and national level epidemic simulations to estimate the medical costs of keeping the US economy open during COVID-19 pandemic under different counterfactual scenarios. We model an unmitigated scenario and 12 mitigation scenarios which differ in compliance behavior to social distancing strategies and to the duration of the stay-home order. Under each scenario we estimate the number of people who are likely to get infected and require medical attention, hospitalization, and ventilators. Given the per capita medical cost for each of these health states, we compute the total medical costs for each scenario and show the tradeoffs between deaths, costs, infections, compliance and the duration of stay-home order. We also consider the hospital bed capacity of each Hospital Referral Region (HRR) in the US to estimate the deficit in beds each HRR will likely encounter given the demand for hospital beds. We consider a case where HRRs share hospital beds among the neighboring HRRs during a surge in demand beyond the available beds and the impact it has in controlling additional deaths.

Interplay of global multi-scale human mobility, social distancing, government interventions, and COVID-19 dynamics

This work quantifies mobility changes observed during the different phases of the pandemic world-wide at multiple resolutions - county, state, country - using an anonymized aggregate mobility map that captures population flows between geographic cells of size 5 km2. As we overlay the global mobility map with epidemic incidence curves and dates of government interventions, we observe that as case counts rose, mobility fell and has since then seen a slow but steady increase in flows. Further, in order to understand mixing within a region, we propose a new metric to quantify the effect of social distancing on the basis of mobility.Taking two very different countries sampled from the global spectrum, We analyze in detail the mobility patterns of the United States (US) and India. We then carry out a counterfactual analysis of delaying the lockdown and show that a one week delay would have doubled the reported number of cases in the US and India. Finally, we quantify the effect of college students returning back to school for the fall semester on COVID-19 dynamics in the surrounding community. We employ the data from a recent university outbreak (reported on August 16, 2020) to infer possible Reff values and mobility flows combined with daily prevalence data and census data to obtain an estimate of new cases that might arrive on a college campus. We find that maintaining social distancing at existing levels would be effective in mitigating the extra seeding of cases. However, potential behavioral change and increased social interaction amongst students (30% increase in Reff) along with extra seeding can increase the number of cases by 20% over a period of one month in the encompassing county. To our knowledge, this work is the first to model in near real-time, the interplay of human mobility, epidemic dynamics and public policies across multiple spatial resolutions and at a global scale.

Evaluating the impact of international airline suspensions on COVID-19 direct importation risk

Global airline networks play a key role in the global importation of emerging infectious diseases. Detailed information on air traffic between international airports has been demonstrated to be useful in retrospectively validating and prospectively predicting case emergence in other countries. In this paper, we use a well-established metric known as effective distance on the global air traffic data from IATA to quantify risk of emergence for different countries as a consequence of direct importation from China, and compare it against arrival times for the first 24 countries. Using this model trained on official first reports from WHO, we estimate time of arrival (ToA) for all other countries. We then incorporate data on airline suspensions to recompute the effective distance and assess the effect of such cancellations in delaying the estimated arrival time for all other countries. Finally we use the infectious disease vulnerability indices to explain some of the estimated reporting delays.